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System and Method For Multiple Channel Statistical Re-Multiplexing 


Inventors: Fang Wu, Sangeeta Ramakrishnan and Ji Zhang 

BACKGROUND OF THE INVENTION 

1. Field of the Invention . 

The present invention relates generally to systems and methods for processing 
compressed bitstreams of data. In particular, the present invention relates to a system and a 
method for multiplexing a plurality of channels for transmission over a single medium. Still 
more particularly, the present invention relates to a system and method for statistical re- 
multiplexing multiple channels. 

2. Description of the Related Art . 

There are presently a variety of different communication channels for transmitting or 
transporting video data. For example, communication channels such as digital subscriber loop 
(DSL) access networks, ATM networks, satellite, or wireless digital transmission facilities are all 
well known. The present invention relates to such communication channels, and for the purposes 
of the present application a channel is defined broadly as a connection facility to convey properly 
formatted digital information from one point to another. A channel includes some or all of the 
following elements: 1) physical devices that generate and receive the signals 
(modulator/demodulator); 2) physical medium that carries the actual signals; 3) mathematical 
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1 schemes used to encode and decode the signals; 4) proper communication protocols used to 

2 establish, maintain and manage the connection created by the channel. The concept of a channel 

3 includes but is not limited to physical channel, but also logical connections established on top of 

4 different network protocols, such as xDSL, ATM, wireless, HFC, coaxial cable, etc. Storage 

5 systems, such as magnetic tapes, optical disks, can also be considered as part of a channel, but 

6 the present invention is not discussed in this context. 

7 The channel is used to transport a bitstream, or a continuous sequence of binary bits used 

8 to digitally represent compressed video, audio or data. The bit rate is the number of bits per 

! J 9 second that the channel is able to transport. The bit error rate is the statistical ratio between the 
Qo number of bits in error due to transmission and the total number of bits transmitted. The channel 

capacity (or channel bandwidth) is the maximum bit rate at which a given channel can convey 
12 digital information with a bit error rate no more than a given value. A video channel or video 

program refers to one or more compressed bit streams that are used to represent the video signal 
\M4 and the associated audio signals. Also included in the video channel are relevant timing, 
Q J multiplexing and system information necessary for a decoder to decode and correctly present the 

16 decoded video and audio signals to the viewer in a time continuous and synchronous manner. 

17 There may be one or more video signals and one or more audio signals per channel. However, in 

18 all realistic cases, each video channel has one video bit stream, together with one or more 

19 compressed audio bit streams. A multiplex is a scheme used to combine bit stream 

20 representations of different signals, such as audio, video, or data, into a single bit stream 

21 representation. In contrast, re-multiplex is a scheme used to combine bit stream representations 

22 of different multiplexed signals into a single bit stream representation. 

23 A digital video signal is a sequence of digitized images that are obtained from the source 
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1 and displayed in the destination in a synchronized manner. Digitized video sequence, when left 

2 in its original digitized form and transmitted over digital communication channels, requires 

3 significant amount of channel bandwidth. The digital video compression techniques, such as 

4 MPEG-1/2/4 and H.26X, can be used to dramatically reduce the channel bandwidth required to 

5 transmit the signal. However, the compression technique also introduces significant 

6 computational complexity into both the encoding and decoding process. Specifically, the 

7 compressed video bit streams, at any given bit rate, cannot be altered again to a different bit rate 

8 without decoding and re-encoding. In addition, the resulting number of bits required to represent 

9 digital video pictures varies from picture to picture and the coded pictures are highly correlated 
\Jo via motion estimation. The problem of delivering real-time digital video bit stream over a 

jl); channel of a given bandwidth becomes a problem of matching the available bandwidth to the 

12 coded video bit stream rate. When the mismatch occurs, re-encoding, or re-compression, must 

lib be done. 

vQ4 Digital Video Compression 

Digital video compression is the two-dimensional signal processing that allows digitized 

16 video frames to be represented digitally in a much more efficient manner. Compression of 

1 7 digital video makes it practical to transmit the compressed signal by digital channels at a fraction 

18 of the bandwidth required to transmit the original signal without compression. International 

19 standards have been created on video compression schemes. These include MPEG-1, MPEG-2, 

20 H.261, H.262, H.263, etc. These standardized compression schemes mostly rely on several key 

21 algorithm schemes as shown in Figure 1 including: motion compensated encoding, transform 

22 coding (DCT transforms or wavelet/sub-band transforms), quantization of the resulting 

23 coefficients, and variable length encoding. The motion compensated encoding 10 removes the 

-3- 

Case 3871 US 


PATENT 

1 temporally redundant information inherent in video sequences. The transform coding 12 enables 

2 orthogonal spatial frequency representation of spatial domain representation of the video 

3 sequence. Quantization 14 of the transformed coefficients reduces the number of levels required 

4 to represent a given digitized video sample and is the major factor in bit usage reduction in the 

5 compression process. The other factor contributing to the compression is the use of variable 

6 length coding (VLC) 16 so that most frequently used symbols are represented by the shortest 

7 code word. In general, the number of bits used to represent a given image determines the quality 

8 of the decoded picture. The more bits used to represent a given image, the better the image 
quality. The hardware or software system that compresses digitized video sequence using the 

M0 above described bit stream schemes is called an encoder or encoding system. In these 

compression schemes, the quantization scheme is a lossy, or irreversible process. Specifically it 

12 results in loss of video textural information that cannot be recovered by further processing at a 

HP later stage. In addition, the quantization process has a direct effect on the resulting bit usage and 

HQ 4 decoded video quality of the compressed bit stream. The schemes in which the quantization 

4i parameters are adjusted control the resulting bit rate of the compressed bit stream. The resulting 

16 bit stream can have either constant bit rate (CBR) or variable bit rate (VBR). ACBR 

1 7 compressed bit stream can be transmitted over a channel that requires the input bit rate to the 

18 channel to be constant over time. Compressed video bit streams are generally intended for real- 

19 time decoded playback at a different time or location. The decoded real-time playback must be 

20 done at 30 frames per second for NTSC standard video and 25 frames per second for PAL 

21 standard video. Thus, all of the information required to represent a digital picture must be 

22 delivered to the destination in time for decoding and display in a timely manner. Therefore, this 

23 requires that the channel must be capable of making such delivery. 
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1 From a different perspective, the transmission channel imposes a bit rate constraint on the 

2 compressed bit stream. In general, the prior art adjusts the quantization in the encoding process 

3 so that the resulting bit rate can be accepted by the transmission channel Because both temporal 

4 and spatial redundancies are removed by the compression schemes and because of variable 

5 length encoding, the resulting bit stream is very sensitive to bit errors or bit losses in the 

6 transmission process compared with transmission of uncompressed video data. In other words, 

7 minor bit error or loss of data in compressed bit stream typically results in major loss of video 

8 quality or in a complete shutdown of operation of the digital receiver/decoder. Furthermore, 

,g 9 real-time multimedia bit streams are highly sensitive to delays. A compressed video bit stream, 

U¥ 0 when transmitted under excessive and jittery delays, causes the real-time decoder buffer to under 

; J |7 flow or overflow, causing the decoded video sequence to be jerky, or causing a loss of 

12 synchronization between the audio and video signals. Another consequence of the real-time 

rU3 nature of compressed video decoding is that lost compressed data will not be re-transmitted. 

Wl4 Because of this sensitivity of compressed bit streams, there is a reluctance to change, modify or 

x *4s re-encode compressed bit streams in the prior art. 

16 Re-Encoding 

17 Re-encoding is the process of performing decoding on an input compressed bit stream 

18 and then encoding back to a compressed bit stream. The prior art includes many ways to apply 

19 rate conversion, or re-encoding, to one or multiple compressed bit streams. Figure 2 shows a 

20 block diagram of a prior art system for transmitting video data over a communication channel 

21 showing the encoding and decoding function in more detail. In particular, as shown, the 

22 encoding includes receiving raw video data and processing the raw video data with motion 

23 estimation 50, transform coding 52, quantization 54, and VLC encoding 54 to produce a 
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1 compressed bit stream. The compressed bit stream can then, because of its reduced size, be 

2 transmitted over any one of a variety of prior art transportation systems 58. The decoding 

3 process is then applied to the compressed bit stream received from the transportation system 58 

4 to obtain the original raw video images. The decoding includes VLC decoding 60, De- 

5 quantization 62, inverse transform coding 64, and motion compensation 66, all in a conventional 

6 manner. 

7 For the purpose of rate conversion in the compressed domain, some exemplary prior art 

8 procedures are shown in Figure 3. For the present invention, re-encoding is defined in its 

9 broadest sense to include partial decoding, recoding, re-quantization, re-transforming, and 

\J0 complete decoding and recoding. Referring now to Figure 3, each of these types of re-encoding 

mi are defined with more particularity. Some of the elements shown may also be needed for 

~~i2 decoding and encoding of the video data. Hence in actual implementation, these common 

S3 elements may be shared between the re-encoder 300 and the decoder/encoder. Partial decoding 

M4 refers to path E where the bit stream is partially decodes system syntax, and video syntax down 

Qi to the picture header to perform frame accurate flexible splicing. Re-coding refers to path D 

16 where variable length encoding and decoding are performed and the DCT coefficients may be 

17 truncated to zero without even going through the inverse quantization steps. This approach 

18 requires the least processing, but in general causes the greatest amount of quality degradation. 

19 Re-quantization refers to path C where variable length encoding, de-quantization, quantization 

20 and decoding are performed but no transform coding is used. The transform coefficients (DCT 

21 coefficients) are requantized before being VLC encoded back. Re-transformation refers to path 

22 B where variable length decoding, de-quantization, inverse transformation, forward transform 

23 coding, quantization and encoding are performed. The video frames are constructed without 
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1 using motion compensation. In the case of B or P pictures, this would mean some of the coded 

2 blocks are motion estimated residual errors. Some form of spatial filtering may be used before 

3 forward transform coding is used in the encoding process. Recoding refers to path A where the 

4 bit streams are complete decoded to raw video and then encoded including the use of motion 

5 estimation and compensation. Each of the paths A, B, C, D, E includes a rate converter for 

6 adjusting the rate of the bit stream to ensure buffer compliance. Each of the rate converters may 

7 be different. For example, the rate converter on path A may be a spatial filter and the rate 

8 converter on path C may perform a quantization step size adjustment while the rate converter on 
[^9 path D performs high frequency elimination. Those skilled in the art will also recognize that the 
I JO components of the re-encoder 300 used (e.g., the path through the re-encoder 300) could also be 
\J11 variably controlled to provide variable bit rate conversion using the re-encoder 300. In various 
'"hi embodiments, the re-encoder 408 may include all, only some or any combination of these 

jlfi components according to which re-encoding, re-quantization, re-transforming and re-coding may 

yQ4 be performed. 

j '35 Generally, motion estimation and compensation is the most computationally expensive; 

16 transform coding and inverse transform coding are also quite expensive. In general, without 

17 special hardware to perform these functions, motion estimation and compensation will take over 

18 80%-90% of the overall decode-encode computation load. The key to a simplified rate 

19 conversion scheme is therefore to bypass some of these expensive steps. For example, in Figure 

20 3, if we take the path B, motion estimation and compensation is avoided. If we take path C, both 

21 motion estimation and compensation and transform coding are eliminated. If we take path D, 

22 quantization steps are also eliminated, in addition to motion estimation and compensation and 

23 transform coding. Of course, path A performs the entire decoding and encoding process, 
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resulting in the most flexibility and potentially the best quality, and is computationally the most 
expensive. 

MPEG-2 Bit Stream Syntax 

Those methods mentioned above can be applied to MPEG-2 program streams, MPEG-1 
streams or other video conferencing compression standards. Thus, while the present invention 
can be applied to any of the various compression technique and is not limited, it will be 
discussed in the present application in the context of MPEG-2 by way of example. This section 
provides brief overview of the MPEG-2 bit stream syntax for better understanding the concepts 
in the invention. 

MPEG-2 compression standard consists of two layers of information. Their relationship 
can be illustrated via Figure 4. The bottom layer is the elementary stream (ES) layer. This layer 
defines how compressed video (or audio) signals are sampled, motion compensated, transform 
coded, quantized, and represented by different variable length coding (VLC) tables. The re- 
encoding of a pre-compressed MPEG-2 bit stream is a process in which the bit stream signal is 
redefined in this layer. 

The next layer is the system layer. The system layer is defined to allow the MPEG-2 
decoder to correctly decode audio and video signals and present the decoded result to the video 
screen in a time continuous manner. The system layer also includes provisions that allow 
unambiguous multiplexing and separation of audio and video compressed signals, and different 
channels of audio and video compressed signals. The system layer consists of two sublayers. 
The first layer is the PES layer; this layer defines how the ES layer bit stream is encapsulated 
into variable length packets, called PES packets. In addition, presentation and decoding time 
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1 stamps are added to the PES packets. There are two different sub-layers above the PES layer, the 

2 transport layer and program system layer. 

3 The transport layer defines how the PES packets are further packetized into fixed sized 

4 transport packet of 1 88 bytes. Additional timing information and multiplexing information are 

5 added to the transport layer. The resulting stream of transport packets is called transport stream. 

6 Transport stream is optimized for use in environments where errors are likely, such as storage or 

7 transmission in lossy or noisy media. Typical applications of transport stream include Direct 

8 Broadcast Service (DBS), digital or wireless cable services, broadband transmission systems, etc. 
% 9 The program system layer defines how the PES packets are encapsulated into variable 
MO size packets. Additional timing and multiplexing information are added to the program system 

| fll layer. The program stream is designed for use in relatively error-free environments and is 

12 suitable for applications that may involve software processing of system information such as 

v\13 interactive multimedia applications. Typical applications of program stream include Digital 

\tl4 Versatile Disks (DVD) and video servers. 

In general a video bit stream can be in elementary stream (ES) format, which means that 

16 no PES, transport or program system layer information is added to the bit stream. The video bit 

17 stream can also be represented in the form of PES stream, transport stream or program stream. 

18 For a given video bit stream, the difference between these different bit streams represented in the 

19 different layers lies in the timing information, multiplexing information and other information 

20 not directly related to the re-encoding process. The information required to perform re-encoding, 

21 however, is contained in the elementary stream layer. The ensuing discussion on re-encoding is, 

22 therefore, not limited to bit streams in any one of the layers. In other words, the discussion on 

23 how to re-encode bit streams in one layer, say in elementary stream layer, can be 
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1 straightforwardly extended to PES stream, transport stream or program streams as well. 

2 With the above background, the system and method for multiple channel statistical re- 

3 multiplexing will now be discussed. 
4 

5 SUMMARY OF THE INVENTION 

6 The present invention is a system and method for multiple channel statistical re- 

7 multiplexing. More particularly, the present invention focuses on methods for manipulating 

8 including recoding of multiple compressed bit streams such that the resulting bit stream rate 
"^9 matches the available channel capacity. Through the use of manipulation, the present invention 

provides loss-less transmission of compressed video bit streams in real-time. The system of the 

\fll present invention includes a plurality of encoders each coupled to a respective channel and 

12 producing a compressed channel, a statistical multiplexer and a transport medium. The statistical 

i multiplexer preferably has a plurality of inputs and an output for combining the input 

nl4 compressed channels into a single output bit stream. The statistical multiplexer further 

•4 5 comprises a plurality of buffers, a plurality of re-multiplexers and a scheduler & multiplexer. 

16 Each of the buffers receives and stores compressed data from a respective encoder for a 

1 7 respective channel. The output of each buffer is coupled to a respective re-multiplexer that re- 

18 encodes the compressed channel in response to control signals from the scheduler & multiplexer. 

19 The scheduler & multiplexer receive the re-multiplexed streams from the re-multiplexers, and 

20 combine them into a single stream that matches the bandwidth of the physical transport medium. 

21 The scheduler & multiplexer control the encoding rate for each of the re-multiplexers thereby 

22 ensuring that when combined, the output matches the channel bandwidth of the transport 

23 medium. The present invention also includes a method for performing statistical re-multiplexing 
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1 including the steps of: performing bit stream analysis; determining a sending rate for each 

2 channel; determining whether the combined bandwidth requirement of all the channels exceeds 

3 the channel capacity; performing rate adjustment by re-multiplexing the channels if the combined 

4 bandwidth requirement of all the channels exceeds the channel capacity; scheduling the channels 

5 for transmission, combining the channels and transmitting the combined channels over the 

6 transport medium. 
7 


8 BRIEF DESCRIPTION OF THE DRAWINGS 

.jg 9 Figure 1 is a block diagram of the general prior art method steps for compressing digital 

video sequences. 

Figure 2 is a block diagram of a prior art method for encoding, transmitting and decoding 

12 digital video sequences or images. 
\iJ3 Figure 3 is a block diagram of prior art method for performing rate conversion or re- 

^Q4 encoding. 

[ 4s Figure 4 is a block diagram graphically representing the layers of the MPEG-2 transport 

16 and program stream protocols. 

1 7 Figure 5 A is a block diagram of preferred embodiment of system including statistical re- 

18 multiplexing before and after transmission according to the present invention. 

19 Figure 5B is a block diagram of preferred embodiment for statistical re-multiplexing 

20 before transmission. 

21 Figure 5C is a block diagram of preferred embodiment for statistical re-multiplexing after 

22 transmission. 
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1 Figure 6 is a block diagram of a preferred embodiment for a video encoder constructed 

2 according to the present invention. 

3 Figure 7 is a block diagram of a preferred embodiment for a receiver constructed 

4 according to the present invention. 

5 Figure 8 is a block diagram of a preferred embodiment for a statistical multiplexer 

6 constructed according to the present invention. 

7 Figure 9 is a block diagram of a preferred embodiment for a re-multiplexer constructed 

8 according to the present invention. 

%9 Figure 10 is a block diagram of a preferred embodiment for a scheduler & multiplexer 

\ d 0 constructed according to the present invention. 

'Stl Figure 1 1 is a block diagram and time line showing the look ahead sliding window 

"i2 technique for bit analysis of the present invention. 

{%13 Figure 12 is a flow chart of a first and preferred method for performing statistical 

vQ4 multiplexing according to the present invention. 

Q5 Figures 13A and 13B are a flow chart of a second embodiment of the method for 

16 performing statistical multiplexing according to the present invention. 

17 

18 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

19 While the present invention will now be described with particularity for the handling of 

20 MPEG-2 digital video compression, those skilled in the art will recognize that the principles of 

21 the present invention may be applied to a variety of other related video compression schemes 

22 such as the H.26X videoconference signals. Specifically, the present application discloses a 

23 technique of performing re-multiplexing to ensure that the bit rate of transmission matches the 
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1 channel capacity. Furthermore, those skilled in the art will recognize that even though a 

2 compressed bit stream typically consists of a multiplex of compressed audio, video and auxiliary 

3 data bit streams, the bit rate reduction process discussed below refers only to the process applied 

4 to compressed video bit streams. 

5 

6 System Overview 

7 Referring now to Figure 5 A, a block diagram of an exemplary transmission system 500 

8 using the present invention is shown. While the present invention is shown in the context of 

9 providing video signal such as in cable television systems, the present invention may be used in 
\J0 any context and this context is provided only by way of example. The present invention is 

LI? 7 directed to use of transcoding to perform what is referred to as "open-loop" statistical 

"i2 multiplexing. In other words, prior art statistical multiplexing requires the close-loop feedback 

ifli between the statistical multiplexing device and the encoding device so that the encoder performs 

m4 the rate adjustment. In contrast, the present invention provides statistical re-multiplexing, with 

Q5 transcoding capability that performs the transcoding. As shows in Figure 5 A, the present 

16 invention may be used in different places or application. Figure 5 A shown a video distribution 

17 system 550 very generally. The system 550 includes a device 503 for upstream multiplexing, a 

18 transport medium 506, a head end 558, a hybrid fiber coaxial (HFC) network 562, and homes or 

19 receivers 564. Those skilled in the art will recognize that this is a general block diagram and 

20 many of the conventional components are not shown of convenience and ease of understanding. 

21 The raw video data is received from one or more sources and provided to the up-stream statistical 

22 re-multiplexing system 503. The up-stream statistical re-multiplexing system 503 provides rate 

23 conversion as necessary to provide a constant bit rate the matches the bandwidth provided by the 
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1 transport medium 506. The up-stream statistical re-multiplexing system 503 will be described in 

2 more detail below with reference to Figure 5B. The output of the up-stream statistical re- 

3 multiplexing system 503 is provided to the transport medium 506, in particular a satellite up link 

4 552. The transport medium 506 preferably comprises the satellite up link 552, a satellite 554, 

5 and a satellite down link 556 as well as other optical couplings of a conventional type as will be 

6 understood to those skilled in the art. The satellite down link 556 is preferably coupled to a cable 

7 television head end 558 in a conventional manner. In addition to the conventional components 

8 of a head end, the present invention provides for addition of a down-stream statistical re- 

;g 9 multiplexing system 560. The down-stream statistical re-multiplexing system 560 is coupled to a 

M0 HFC network 562 and provides rate conversion as necessary to provide a constant bit rate the 

\ J]7 matches the bandwidth provided by the HFC network 562. The down-stream statistical re- 

12 multiplexing system 560 will be described in more detail with reference to Figure 5C below. 
:||3 Referring now to Figure 5B, a preferred embodiment of the up-stream statistical re- 

\JJ4 multiplexing system 503 constructed according to the present invention and including statistical 

Qs re-multiplexing is shown. The system 503 is adapted for use on an MPEG-2 transport stream. 

16 The preferred embodiment of the system 503 preferably comprises a plurality of video channels, 

1 7 CHl-CHrc, providing video sequences, a plurality of video encoders 502a-n, a statistical re- 

18 multiplexer 504, and a up-link converter 505. Each of the plurality of video encoders 502a-n is 

19 coupled to receive a respective one of the plurality of video channels, CH1-CH«. Each channel 

20 CHl-CHw preferably provides a non-compressed stream of video sequences that is input to a 

21 respective video encoder 502a-n. Each video encoder 502a-n in turn compresses the received 

22 video sequences to produce a compressed bit stream. The statistical re-multiplexer 504 

23 preferably has a plurality of inputs for receiving the compressed video sequences and for 
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1 combining the input compressed channels into a single output bit stream provide at its output. 

2 The statistical re-multiplexer 504 further comprises a plurality of de-multiplexers 802a-n and a 

3 scheduler & multiplexer 804, as will described in more detail below with reference to Figure 8. 

4 The statistical re-multiplexer 504 receives and stores compressed data for each respective 

5 channel The statistical re-multiplexer 504 re-encodes the compressed channel in response to 

6 control signals from the scheduler & multiplexer 804 and combines the re-encoded channels into 

7 a single bit stream of output over the transport medium 506. The statistical multiplexing is 

8 controlled by the scheduler 804 that determines the order of the programs in which to transmit 
IS 0 the packets. The bit rate of the output of the statistical multiplexer 504 preferably matches the 
M0 bandwidth of the physical transport medium 506. The scheduler & multiplexer 804 control the 
j% output thereby ensuring that the output matches the channel bandwidth of the transport medium 

, ; h 506. The up-link converter 505 preferably couples the output of the statistical re-multiplexer 504 

|1|5 to the transport medium 506. The up-link converter 505 preferably formats the output of the 

statistical re-multiplexer 504 so that it may be sent over the transport medium 506 in a 

: % conventional manner. The up-link converter 505 in alternate embodiments may be part of the 

1 6 transport medium 506. 

1 7 Referring now to Figure 5C, a preferred embodiment of the down-stream statistical re- 

18 multiplexing system 560 constructed according to the present invention and including statistical 

19 re-multiplexing is shown. The system 560 is again adapted for use on an MPEG-2 transport 

20 stream. The preferred embodiment of the system 560 preferably comprises a down link 

21 converter 507, a splitter 509, and a statistical re-multiplexer 504. The down link converter 507 

22 coupled the splitter 509 to the transport medium 506 and coverts the satellite transmission into a 

23 form usable by the splitter 509 in a conventional manner. Similarly, the splitter 509 receives a 

-15- 

Case 3871 US 


PATENT 

1 signal from the transport medium 506 and demodulates, splits and converters it into a plurality of 

2 signals on different channels. Thus, the splitter 509 has an input and a plurality of outputs. Each 

3 of the output of the splitter 509 provides a compressed sequence of video signals. The statistical 

4 re-multiplexer 504 is preferably the same as described above and used in the down-stream 

5 statistical re-multiplexing system 503 , and will be described below in more detail with reference 

6 to Figure 8. The output of the statistical re-multiplexer 504 is coupled to the HFC network 

7 562.The same statistical multiplexed signal consisting of signals for all programs is broadcast to 

8 each of the receivers 508a-n. The HFC network 562is in turn coupled to each of the plurality of 
^pj 9 receivers 508a-n. Each of the plurality of receivers 508a-n receives the entire bit stream from the 
\J0 statistical multiplexer 504, extracts the program content destined for other devices (not shown) 
[Ml coupled to the receiver 508a-n and decodes the compressed video signals. 

t2 Referring now to Figure 6, a preferred embodiment of a video encoder 502 is shown. The 

III J video encoder 502 preferably comprises: a video scene analyzer 602, a rate controller 604, and a 

VJ34 compressor 606, The video scene analyzer 602 has an input and a first and second output. The 

Wj video scene analyzer 602 performs analysis on the video sequence to determine the complexity 

16 of the sequence as it is related to the video compression process. In other words, it provides 

1 7 information to the compressor as to how many bits are to be used to encode a particular video 

18 image. Typical measures used by the analyzer include: amount of motion, amount of texture, 

19 progressive or interlaced scanning format (telecine), scene changes and fades, as well as the 

20 amount of noise. The input of the video scene analyzer 602 is coupled to an input line to receive 

21 the uncompressed video data for a given channel. The first output of the video scene analyzer 

22 602 is coupled to the rate controller 604 to provide information used by the rate controller to 

23 determine the amount and type of compression that the compressor 606 should perform. Typical 
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1 information exchanged between the rate controller 604 and the video scene analyzer 602 

2 includes: video complexity measure, scene cut location, fade location, progressive or interlaced. 

3 The second output of the video scene analyzer 602 is coupled to the input of the compressor 606, 

4 The second output provides the video data and relevant timing information so that it can be used 

5 by the compressor 606 to provide a bit stream at a desired bit rate. The compressor 606 similarly 

6 has a data input, a data output and a control input/output. The data input of the compressor 606 is 

7 coupled to the output of the video scene analyzer 602, the data output forms the output of the 

8 encoder 502. The compressor 606 compresses the bit stream using the conventional methods 

; g 0 described above including but not limited to motion compensated encoding, transform coding 

\J0 (DCT transforms or wavelet/sub-band transforms), quantization of the resulting coefficients, and 

\ fil variable length encoding. The compressor 606 is preferably able to perform any of the various 

12 types of compression in response to control signals from the rate controller 604. A compressor 

j3 3 606 is typically a real-time encoder. In the case of MPEG-2 encoding, the compressor 606 may 

} ^ i; 

\§4 also include audio compressor that generates compressed audio signals (preferably in AC-3 or 

Qj MPEG format). The compressor 606 may generate the signals either in transport stream format 

16 (packets of 1 88 bytes), packetized elementary streams (PES) of variable length packets, or 

17 elementary streams (ES) of variable length packets. Other possible implementations of the 

18 compressors include software processing on general-purpose computers. The compressor may 

19 also include other types of video compression (such as H.26X or wavelet). The compressor 606 

20 is coupled to the rate controller 604 to receive signals indicating the type of compression to be 

21 performed and preferred bit rates. Specifically, the rate controller 604 communicates with the 

22 compressor 606 to determine the optimal allocation of bits (called bit budgets) on a per coded 

23 picture basis. The compressor 606 in turn sends to the rate controller 604 the actual number of 
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1 bits used to code each video image. The two components (604 and 606) exchange information 

2 each time a video picture is coded. The rate controller 604 has a plurality of ports and one port is 

3 coupled to the compressor 606, one to the video scene analyzer 602 and one to receive rate 

4 control information. The rate controller 604 specifies a bit rate for the bit stream output by the 

5 compressor 606. The rate controller 604 receives data from the video scene analyzer 602, 

6 specifies what bit rates are possible, and in response to signals from the statistical multiplexer 

7 504 provides commands to the compressor 606 to perform compression that will achieve a 

8 desired bit rate. This can be done by any source (not shown) to control the video encoding 
;g 9 (compression). However, in an alternate embodiment, the statistical re-multiplexer 504 sends 

\ J 0 messages to the rate controller 604 including targeted bit usage for the next coded pictures for all 

| J] 7 of the channels to be multiplexed. The rate controller 604 may also preferably send messages to 

12 the re- statistical multiplexer 504 as to the number of bits used for the coded pictures before the 

v^3 associated data is queued into the statistical multiplexer buffer. 

\jl4 Referring now to Figure 7, a preferred embodiment of a receiver 508 is shown. The 

Qs receiver 508 preferably comprises: a de-multiplexer 702, a buffer 704 and a decoder 706. The 

16 de-multiplexer 702 has an input and an output, and preferably has its input coupled to the 

1 7 transport medium 506. Generally, the de-multiplexer 702 filters out all data not destined for this 

18 receiver. More specifically, the de-multiplexer 702 receives the bit stream sent by the statistical 

19 multiplexer 504 and extracts a video elementary stream payload from the video transport stream. 

20 The de-multiplexer 702 essentially performs the inverse function of the statistical multiplexer 

21 504 but to a more limited extent. While the statistical multiplexer 504 combines re-multiplexed 

22 streams into a single stream, the de-multiplexer 702 receive the single stream and extracts the re- 

23 multiplexed channel corresponding to (or addressed to) the particular receiver 508 and discards 
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1 the other information. The output of the de-multiplexer 702 provides the extracted re- 

2 multiplexed stream. The output of the de-multiplexer 702 is coupled to the input of a buffer 704. 

3 The buffer 704 is filled with packets destined for this particular decoder 706 at a rate determined 

4 by the channel transmission. The buffer 704 provides for temporary storage of the bit stream 

5 before decoding. The buffer 704 ensures that there is enough scenes from the compressed bit 

6 stream to perform decoding. Whether the buffer 704 over flows or under flows is dictated by the 

7 scheduling of channels (insertion of channel data) in the single bit stream by the statistical 

8 multiplexer 504. The output of the buffer 704 is coupled to the input of a decoder 706. The 

[% 9 decoder 706 is any one of a conventional type as will understood by those skilled in the art and 

\J0 performs the inverse function of the compressor 606. Those skilled in the art will recognize that 

j HI although only one decoder 706 is shown for each receiver 508, in reality, there may be multiple 

112 decoders 706 inside each receiver 508 because audio and video require different decoding 

■j|b processes. The output of the decoder 706 forms the output of the receiver 508 and provides the 

\J14 uncompressed video data ready for use. 
Qs 

16 Statistical Re-Multiplexer 

1 7 Statistical re-multiplexing is a technique that simultaneously analyzes and performs 

18 necessary re-encoding with multiple channels of video signals and combines the resulting bit 

19 streams into a single bit stream for transmission. Statistical re-multiplexing explores the variable 

20 rate nature of the compressed video bit streams and the statistical nature of such bit rates. 

21 Specifically, it combines multiple channels of variable bit rate (VBR) bit streams of compressed 

22 video so that the resulting multiplex has a constant fixed rate. This technique, when properly 

23 implemented, results in significant bandwidth savings when transmitting multiple channels of 
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1 pre-compressed digital video signals. The key difference from statistical multiplexing, therefore, 

2 is that the inputs to the statistical re-multiplexer 504 are pre-compressed bit streams. The re- 

3 scheduling of packets, re-encoding, together with the rate control and re-multiplexing forms the 

4 functionality of the statistical re-multiplexer 504 as will be described more detail below. The re- 

5 encoding, together with the rate control and re-multiplexing performs the same functions as that 

6 of statistical multiplexing. While the present invention will now be described with reference to 

7 re-multiplexing, those skilled in the art will recognize that the statistical re-multiplexer 504 may 

8 be used alone on the compressed video streams eliminating the need for the encoders 502 of 
]n9 Figure 5B. In this case, the statistical re-multiplexer 504 performs re-coding, selective or 

1^0 complete re-coding, on all of the participating channels so that the multiplexed output bit stream 

\*1 1 has a given constant bit rate. Statistical re-multiplexing can be considered as the combination of 

12 selective re-coding and channel hopping. 

jib Referring now to Figure 8, the statistical multiplexer 504 is shown in more detail. Figure 

134 8 provides a statistical multiplexer 504 in an example where 4 channels (or programs) are 

'45 statistically re-multiplexed. While the present invention shows only 4 channels being re- 

16 multiplexed, those skilled in the art will recognize that any number of programs or channels may 

17 be multiplexed. The statistical re-multiplexer 504 further comprises a plurality of de- 

18 multiplexers 802a-n, preferably one for each channel, and a scheduler & multiplexer 804. Each 

19 of the de-multiplexers 802a-n is coupled to receive a particular compressed channel and has a 

20 data output coupled to the scheduler & multiplexer 804 by line 824. The scheduler & 

21 multiplexer 804 has the functionality described above and is also coupled to a control input of 

22 the de-multiplexers 802 by line 822. The de-multiplexers 802 are described in more detail below 

23 with reference to Figure 9. In an alternate embodiment (not shown), the scheduler & multiplexer 
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1 804 also has a second control input coupled by lines to respective encoders 502. The output of 

2 the scheduler & multiplexer 804 provides a bit stream that matches the channel capacity. 

3 Referring now to Figure 9, a preferred embodiment of a re-multiplexer 802 is shown. 

4 Each of the re-multiplexers 802 preferably comprises: a bit stream analyzer 902, a rate controller 

5 904, and a re-encoder 906. The bit stream analyzer 902 has an input and a first and second 

6 output. The bit stream analyzer 902 parses the bit streams to determine the bit usage of each of 

7 the channels for some pre-determined amount of time, T seconds, ahead of what is currently 

8 being multiplexed and sent over the transport medium 506. The input of the bit stream analyzer 
'^9 902 is coupled to an input line to receive the compressed video data for a given channel from a 
!*?0 respective video encoder 502. The first output of the bit stream analyzer 902 is coupled to the 

rate controller 904 to provide information used by the rate controller to determine the amount 

12 and type of compression that the re-encoder 906 should perform. The bit stream analyzer 902 is 

ill 3 different from video scene analyzer 602 in that here the bit stream analyzer 902 operates in the 

^S4 compressed digital domain. In other words, the bit stream analyzer 902 inspects the input 

-4 5 compressed video bitstreams to extract information that can be used to assist the re-encoding 

16 process. The information may include: number of bits used for each of the coded pictures, 

17 picture coding type, average quantizer scale value for each of the coded pictures, whether the 

18 coded picture is coded due to fade or scene cuts in the original video sequence, etc. The second 

19 output of the bit stream analyzer 902 is coupled to the input of the re-encoder 906. The second 

20 output provides the video data and relevant timing information so that it can be used by the re- 

21 encoder 906 to provide a bit stream that is further compressed to have a desired bit rate. 

22 The re-encoder 906 similarly has a data input, a data output and a control input/output. 

23 The data input of the re-encoder 906 is coupled to the output of the bit stream analyzer 902, and 
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1 the data output forms the output of the re-multiplexer 802. The re-encoder 906 compresses the 

2 bit stream using the conventional methods described above including but not limited to motion 

3 compensated encoding, transform coding (DCT transforms or wavelet/sub-band transforms), 

4 quantization of the resulting coefficients, and variable length encoding. The re-encoder 906 is 

5 preferably able to perform any of various type of compression in response to control signals from 

6 the rate controller 904. The preferred embodiment for the re-encoder 906 can be either path A or 

7 B or C in Figure 3. The re-encoder 906 also is able to affect rate changes in the cases where 

8 excessive rate reduction is required by dropping B-frames and repeating the previous frame. The 
'/q9 re-encoder 906 is coupled to the rate controller 904 to receive signals indicating the type of 

(Jo compression to be performed and preferred bit rates. The information exchanged between the 

rate controller 904 and the re-encoder 906is very similar to that of the rate controller 604 and the 

12 compressor 606. 

jip The rate controller 904 has a plurality of ports and one port is coupled to the compressor 

;XM 660, one to the bit stream analyzer 902 and one to the statistical multiplexer 504. The rate 

Qs controller 904 specifies a bit rate for the bit stream output by the re-encoder 906. The rate 

16 controller 904 receives data from the bit stream analyzer 902, specifies what bit rates are 

17 possible, and in response to signals from the scheduler & multiplexer 804 provides commands to 

18 the re-encoder 906 to perform compression that will achieve a desired bit rate 

19 The statistical multiplexer 504 of the present invention is designed to ensure that the 

20 video bit-stream, when transmitted over the transport medium 506 to the destination, does not 

21 cause the decoder buffer 704 to under flow or overflow. Of course, the audio signals must also 

22 be properly scheduled so that decoder buffer 704 for audio does not overflow and under flow. 

23 Furthermore, the destination decoder 706 needs to maintain audio video synchronization at all 
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1 times. These constraints can be described mathematically using the following definitions and 

2 assumptions. First, (z) is used to represent channel i, and there are total of / video programs. 

3 Second, C is the total transmission channel capacity, in number of bits per second. Finally, R(i,t) 

4 is the bit rate used to transmit channel i at time t. With this notation, the statistical multiplexer 

5 504 must satisfy the equation £ R(U t)ZC for all t. In other words, at any time /, the total 

6 transmission rate out of the statistical multiplexer 504 must not exceed the channel capacity. In 

7 addition, if R(i, t) is the sending bit rate for the statistical multiplexer 504 to send the bit stream 
^ 8 for channel C at the given time t; R^z, t) is the minimum bit rate required to transmit the bit 

: J P stream data for a channel without under flowing the decoder buffer 706 At the given time t, 

M° RmaxO'. 0 is the maximum bit rate to transmit the bit-stream data channel (z) without overflowing 

'=41 the decoder buffer, then to avoid decoder buffer over or under flow, we must 

|Lp have R mh,(z',/) < R(i, t)<R m*(i,t) for all t . The scheduler & multiplexer 804 is designed to 

i=|5 ensure that this equation is satisfied for all channels. 

Q4 Referring now to Figure 10, a preferred embodiment for the scheduler & multiplexer 804 

15 is shown. The scheduler & multiplexer 804 preferably comprises a controller 1002, a scheduling 

16 table 1004, a multiplexer 1006, a FIFO buffer 1008, a plurality of input FIFO buffers lOlOa-n, 

17 and a filler packet adder or inserter 1012. The controller 1002 is coupled to control each of the 
75 re-multiplexers 802a-n by signal lines 822a-n. Each of the input signal lines 824a-824n is 

19 coupled to the input of a respective input FIFO buffer 1 01 Oa-n. The input FIFO buffers 1 0 1 Oa-n 

20 store the data from the re-multiplexers 802a-n temporarily until it can be scheduled for output 

21 through the multiplexer 1006. The input FIFO buffers 101 Oa-n are first-in, first-out buffers of a 

22 conventional type. The controller 1002 is also coupled to the scheduling table 1004. The 
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1 scheduling table 1 004 provides priority data indicating what streams or programs have priority 

2 and the amount of bandwidth of the transport channel at each priority level. The scheduling table 

3 1004 is also controls the multiplexer in response to signals from the controller 1002. The 

4 scheduling table 1004 in one embodiment uses a single table. In yet another embodiment, the 

5 scheduling table 1004 is a plurality of fixed size tables that can be alternatively used. With such 

6 a configuration, one table may be used for scheduling while another is being modified. The 

7 controller 1002 receives and uses input from the scheduling table 1004 to determine the amount 

8 of re-encoding necessary for each program and the order for sending the programs over the 

9 transport medium 506 to meet the constraints noted above. The scheduling table 1004 has 

I *J 0 control outputs coupled to the multiplexer 1 006 and the FIFO buffer 1 008 via the filler packet 

\H1 inserter 1012. The scheduling table 1004 generates control signals to multiplexer 1006. The 

H 2 multiplexer 1 006 pulls the data from the particular FIFO 1 0 1 Oa-n according to the command 

'^3 from the scheduling table 1004, and sends the data to the input of the FIFO buffer 1008. The 

$4 FIFO buffer 1008 is clocked at a given constant bit rate which meets the capacity of channel 506. 

Q5 The FIFO buffer 1008 allows the re-multiplexed inputs to be mixed, ordered and sent in various 

1 6 ways simply by varying the order in which the respective streams are stored in the FIFO buffer 

17 1 008 . The scheduling table 1 004 is also coupled to the filler packet inserter 1 0 1 2, and the filler 

18 packet inserter 1012 is in turn coupled to an input of the FIFO buffer 1008. Since the scheduler 

19 & multiplexer 804 outputs data at a constant bit rate equal to the bandwidth of the transport 

20 medium 506, the filler packet inserter 1012 is used to add null packets for the times when the 

21 compressed on line 824a-n are insufficient to maintain the constant bit rate. Since the scheduling 

22 table 1004 knows the amount of data in the FIFO buffer 1008 at any given time, it can activate 

23 the filler packet inserter 1012 as necessary to insert extra place holder packets. The extra place 
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1 holder packets may be null packets or packets containing stuffing bytes. In an alternate 

2 embodiment, the filler packet inserter 1012 could also insert user defined program identification 

3 numbers (PIDs) and user information or that could be used by the cable transport plant later 

4 down stream. 

5 Referring now to Figure 12, the operation of the statistical re-multiplexer 504, in 

6 particular, the controller 1002 will better be understood. The statistical re-multiplexer 504 and 

7 its constituent components perform the steps to ensure that the bandwidth of the transport 

8 medium 506 is fully utilized. The process has four basic steps including: (1) performing bit 
;gP stream analysis; (2) determining a sending rate for each channel; (3) performing rate adjustment; 
I d 0 and (4) scheduling the channels for transmission. 

[Ml The process begins with step 1204 by performing bit stream analysis. In order to 

~i2 determine at what rate each channel must be sent, the statistical re-multiplexer 504 analyzes the 

j |5 bit streams ahead of time, by adopting look-ahead windowing technique. The term "look-ahead 

\%4 window" refers to that the statistical re-multiplexer 504 parses the bit streams to determine the 

1 35 bit usage of each of the channels for Tj seconds ahead of what is currently to be sent, in order to 

1 6 decide the incoming and outgoing bit rate for each channel. Referring also to Figure 1 1 , a 

/ 7 graphic representation of this process is shown. Having made an examination, the statistical re- 

/ 8 multiplexer 504 may decide to send out number of bits corresponding to a time T, for each of the 

19 channels, where T, may be less than or equal to T. For the case T, < T, the statistical re- 

20 multiplexer 504 examines the input bit streams from all different programs in an interval T, and 

21 only sends out T l part of the data. In the next iteration, the statistical re-multiplexer 504 

22 examines the data after T„ and the examining data period still uses a time window size of T. 

23 This scheme is referred to as a sliding window technique. A typical choice for T, would be T/2. 
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This also means buffering of data up to T seconds would be required, and a variable time period 
T with a fixed mean value is a good and unique approach to analyzing the bit stream. 

The process continues in step 1208, where the method determines an incoming bit rate for 
each channel, the incoming bit rate is max( (bits_in_T/T), (bits_in_T 1 /T I ) ). 

Next in step 1210, the method tests whether the combined bandwidth requirement of all 
the channels is equal to the channel capacity. This is tested by comparing the total bandwidth 
requirements of all the channels to the channel capacity over the time window under 
consideration. If the combined bandwidth requirement of all the channels is equal to the channel 
capacity, the method proceeds directly to step 1214. In step 1214, the statistical re-multiplexer 
504 schedules all the data for all the channels into a set of schedule tables that give the guidelines 
for what time and what PID of data will be sent out. 

On the other hand, if the combined bandwidth requirements of all the channels do not 
equal the channel capacity, then the method continues in step 1212 to perform rate adjustment by 
re-multiplexing. In the case where the bandwidth requirements exceed the channel capacity, then 
the statistical re-multiplexer 504 decreases the sending rates for channels whose rate must be 
greater than the minimum rate needed to avoid under flow. The choice of which channel to 
reduce the sending rate is based on the difference between the current transmission rate and the 
minimum rate which will guarantee this channel not under flow. It is preferred to reduce the rate 
of a channel whose rate difference is higher than others. Having attempted to do that for all the 
channels if the bandwidth requirements are still not met (i.e., the total transmission rate still 
exceeds the channel capacity), the statistical re-multiplexer 504 then performs rate reduction on 
selected channels in order to meet the channel requirements. 
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1 When the sending rate cannot be further decreased for any of the channels by the 

2 statistical multiplexer 504 without under flowing at least one decoder buffer, the statistical re- 

3 multiplexer 504 performs rate reduction on some or all of the channels in step 1212. As for the 

4 uncompressed data case, it is even easier, the statistical re-multiplexer 504 just communicates 

5 with the encoder and demands the proper bit rate from the encoder. Again on each channel that 

6 is rate converted some or all of the frames in the window under consideration may be rate 

7 converted. The decision as to which channel to be rate converted and how much to be reduced 

8 depends on the statistics from each video channel such as the complexity measurement, the 
picture type, the picture size, GOP size, and etc, and it also depends on the QOS service 

\2o parameter user entered. The preventive rate reduction is also adopted to ensure the flexibility of 
l&l choosing a particular channel, which does not affect the visual result adversely. Having parsed 

~i2 the bit stream for each of the channels for T seconds ahead, the statistical multiplexer 504 uses 

||5 the information obtained, to determine how much rate conversion to perform. Pictures with 

i jjjf smaller complexity by a certain measurement can undergo a larger percentage of rate reduction 

: 3j with a smaller loss in picture quality. In general, rates can be reduced on B-frames with lesser 

16 impact on overall picture quality, and pictures, which occur later in the Group of Pictures (GOP), 

17 can be rate reduced to a larger extent with a smaller impact on picture quality compared to 

18 pictures earlier in the GOP. When the degree of the rate reduction is so significant that the 

19 conventional rate reduction method cannot achieve that reduction rate, selective B frames will be 

20 dropped, and the previous frame will be repeated to replace the dropped B frame to make sure 

21 the program interval does not change. 

22 In case the bandwidth requirements are less than the channel capacity the statistical re- 

23 multiplexer 504 then increases the sending rate for those channels whose rate is below the 
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1 maximum rate. If all the sending rates for all channels are at the maximum rate yet the total 

2 sending rate is still below the channel capacity, a null packet or filler data is inserted until the 

3 total bandwidth equals the channel capacity. An increase in sending rate is achieved by 

4 expanding or contracting the time window under consideration for each channel on an individual 

5 basis. By sending the same amount of bits in a smaller time window or sending more bits in the 

6 same time window achieves the effect of increasing the sending rate. A decrease in sending rate 

7 is achieved by expanding or contracting the time window under consideration for each channel 

8 on an individual basis. In other words, by sending the same amount of bits in a bigger time 

; 9 window or sending lesser bits in the same time window achieves the effect of decreasing the 

I Jo sending rate. After performing step 1212, the method returns to step 1210 to determine whether 

ill 1 the bandwidth requirements no match the channel capacity. 

y 2 m step 1214, the method schedules the programs or channels for delivery. Once the 

j|i sending rates for each of the channels have been determined in step 1208, the statistical 

i|4 multiplexer 504 schedules the individual packets of each of the channels. The packets are 

=35 scheduled such that the entries in the scheduling table are evenly distributed across the different 

16 channels. The number of packets to be scheduled for each channel is determined by the rate for 

1 7 that channel, total bandwidth, and the size of the schedule table. It is computed using the 

18 equation: 

19 PacketPerTable (I) - (ScheduleTableSize * Rate(i))/(TotalBandwidth) 

20 Once the number of packets to be allocated has been computed, the statistical re- 

21 multiplexer 504 schedules the individual packets for each channel. Any slots in the schedule 

22 table that are not assigned to any of the channels are filled with user controllable filler packets 

23 that could be null packets in default, or the opportunity data packets. The same method can also 
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1 be used to schedule non- video packets. Once the packets have been scheduled, the packets for 

2 the programs are combined and transmitted over the medium in step 1216. 

3 Referring now to Figure 13A and 13B, a second embodiment of the method for 

4 performing statistical multiplexing according to the present invention. The method begins in 

5 step 1302 by examining the bit stream. Next in step 1304, the method tests whether the sum of 

6 the bit rates for all channels is equal to the channel capacity. If so, the method continues in step 

7 1 306, to schedule the channel for transmission as described above, and then transmits the 

8 channels over in step 1308. 

If the sum of the bit rates for all channels is not equal to the channel capacity, the method 

\A0 transitions from step 1304 to step 1310. In step 1310, the method determines whether the sum of 

i Hi the bit rates for all channels exceeds the channel capacity. If so, the method continues in step 

1-2 1 3 12 to select a channel, and in step 13 14 the method tests whether the channel rate is greater 

|3|3 than the minimum bit rate for the channel. If the channel rate is greater than the minimum bit 

igtf rate for the channel, then the method performs step 1316 before continuing to step 1318. In step 

1 15 1 3 1 6, the method reduces the bit rate for the selected channel to the minimum bit rate for the 

16 channel. If the channel rate is not greater than the minimum bit rate for the channel, then the 

1 7 method moves directly to step 1318. In step 1318, the method tests whether there are other 

18 channels with channel rates greater than their respective minimum bit rates. If so, the method 

19 loops back to step 1304 to reduce the bit rates of those channels. If not, the method continues to 

20 step 1320 to perform rate conversion on the selected channel. After step 1320, the method 

21 moves to step 1304 to determine if the channels are ready for transmission or require additional 

22 bit rate reduction. 

23 If the sum of the bit rates for all channels is not greater than the channel capacity, the 
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method moves to step 1322 of Figure 13B. In step 1322, the method determines whether the sum 
of the bit rates for all channels is less than the channel capacity. If so, the method continues in 
step 1324 to select a channel, and in step 1314 the method tests whether the channel rate is less 
than the maximum bit rate for the channel. If the channel rate is less than the maximum bit rate 
for the channel, then the method performs step 1328 before continuing to step 1330. In step 
1328, the method increases the bit rate for the selected channel to the maximum bit rate for the 
channel. If the channel rate is not less than the maximum bit rate for the channel, then the 
method moves directly to step 1330. In step 1330, the method tests whether there are other 
channels with channel rates less than their respective maximum bit rates. If so, the method loops 
back to step 1304 to increase the bit rates of those channels. If not, the method continues to step 
1332 to add null packets or opportunity data packets on the selected channel. After step 1332, 
the method moves to step 1304 to determine if the channels are ready for transmission or require 
additional bit rate reduction. 

While the present invention has been described with reference to certain preferred 
embodiments, those skilled in the art will recognize that various modifications may be provided. 
These and other variations upon and modifications to the preferred embodiments are provided for 
by the present invention, which is limited only by the following claims. 
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WHAT IS CLAIMED IS : 

1 . A device for performing statistical re-multiplexing of digital video signals, the 
device comprising: 

a scheduler having first and second data inputs, a data output, and first and second control 
ports, the scheduler for combining signals at the plurality of data inputs into a 
signal at the data output that matches channel bandwidth of a transport medium 
coupled to the output of the scheduler; 

a first de-multiplexer having a data input, a control port and a data output, the first de- 
multiplexer re-coding an input signal in response to a first control signal received 
at the control port, the data input of the first de-multiplexer coupled to receive a 
first channel of data, the control port of the first de-multiplexer coupled to the first 
control port of the scheduler, the data output of the first de-multiplexer coupled to 
the first data input of the scheduler; and 

a second de-multiplexer having a data input, a control port and a data output, the second 
de-multiplexer re-coding an input signal in response to a second control signal 
received at the control port, the data input of the second de-multiplexer coupled to 
receive a second channel of data, the control port of the second de-multiplexer 
coupled to the second control port of the scheduler, the data output of the second 
de-multiplexer coupled to the second data input of the scheduler. 

2. The device of claim 1 further comprising an encoder having a data input, a data 
output, and a control port, the encoder for performing compression and rate adjustment on a data 
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stream, the data input of the encoder coupled to receive a non-compressed stream of video 
sequences, the output of the encoder coupled to the input of the first de-multiplexer. 

3. The device of claim 1 further comprising a splitter having a data input, first and 
second data output, the splitter converting a single stream into a plurality of streams of data, the 
input of the splitter coupled to receive a stream of compressed data having a plurality of 
channels, the first output of the splitter coupled to the data input of the first de-multiplexer, and 
the second output of the splitter coupled to the data input of the second de-multiplexer. 

4. The device of claim 1 further comprising n additional de-multiplexers, each of the 
n additional de-multiplexers having a data input, a control port and a data output, each of the n 
additional de-multiplexers re-coding an input signal in response to a signal received at the 
respective control port, and wherein the scheduler has a corresponding data input and control port 
for each of the n additional de-multiplexers, the data input of each of the n additional de- 
multiplexers coupled to a respective corresponding data input of the scheduler, the control port of 
each of the n additional de-multiplexers coupled to a respective corresponding control port of the 
scheduler. 

5 . The device of claim 1 , wherein the scheduler further comprises: 

a scheduling table having an first input and a first output for generating control signals to 
select a particular channel for output based on an input control signal, the first 
input coupled to the first and second de-multiplexer; and 

a multiplexer having a plurality of data inputs, a data output, and a control input for 
providing at the data output of the multiplexer one of signals received at the 
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7 plurality of data inputs, the control input of the multiplexer coupled to the output 

8 of the scheduling table, each of the plurality of data inputs coupled to the 

9 respective data outputs of the first and second de-multiplexer. 

1 6. The device of claim 5, wherein the scheduling table is a plurality of fixed sized 

2 tables that can be alternatively accessed. 

1 7. The device of claim 5, wherein the scheduler further comprises: 

2 a controller for controlling a rate at which the scheduler outputs data and which channels 
O 3 are output, the controller having a first set of control ports, a first control port and 

a second control output, at least one port in the first set of control ports coupled to 

!fl 5 the control input of the first de-multiplexer and the control input of the second de- 

'-4 6 multiplexer, and the first control output coupled to the first input of the scheduling 

P7 table; 

j j<? a first buffer having a data input, a data output, and a control input, for storing data on a 

lEj 9 first in, first out basis, the data input of the first buffer coupled to the data output 

10 of the first de-multiplexer, the data output of the first buffer coupled to a first of 

11 the plurality of data inputs of the multiplexer, and the control input coupled to the 

12 second control output of the controller; and 

13 a second buffer having a data input, a data output, and a control input, for storing data on 

14 a first in, first out basis, the data input of the second buffer coupled to the data 

15 output of the second de-multiplexer, the data output of the second buffer coupled 

16 to a second of the plurality of data inputs of the multiplexer, and the control input 

1 7 coupled to the second control output of the controller. 
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1 8. The device of claim 5, wherein the scheduler further comprises: 

2 a filler packet inserter having an input and an output for providing filler packet data in 

3 response to a control signal on the input, the input of the filler packet inserter 

4 coupled to a second control output of the scheduling table; and 

5 an output buffer having a first input, a second input and an output, for storing data, the 

6 first input of the output buffer coupled to the output of the multiplexer and the 

7 second input of the output buffer coupled to the output of the filler packet inserter. 

% J 9. The device of claim 8, wherein the filler packet inserter inserts a packet that 

22 contains a user-defined program identification number (PID) as the filler packet. 

4 ] 10- The device of claim 5, wherein the controller receives and analyzes data from the 

\J2 first and second de-multiplexers for a sliding window of time T, and then sends out number of 

«5 bits corresponding to a time T x for each of the channels, where T x is less than or equal to T. 

~/ 1 1 . A method for processing video data, the method including the steps of: 

2 performing bit stream analysis; 

3 determining an incoming bit rate for each channel; 

4 determining whether the combined bandwidth requirement of all the channels exceeds the 

5 channel capacity; 

6 performing rate adjustment by re-multiplexing the channels if the combined bandwidth 

7 requirement of all the channels does not equal the channel capacity; 

8 scheduling the channels for transmission; and 

9 combining the channels and transmitting the combined channels over the transport 
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10 medium. 

/ 12. The device of claim 1 1, wherein the step of performing bit stream analysis is 

2 performed for a sliding window of time having duration T, and wherein the step of combining 

3 the channels and transmitting the combined channels is for a time T l for each of the channels, 

4 where time T\ is less than or equal to T. 

1 13. The device of claim 1 1, wherein the step of performing rate adjustment includes 

2 inserting extra packets, and the step of combining and transmitting includes adding and 
1 3 3 transmitting the inserted extra packets. 

W 14. The device of claim 1 3, wherein the extra packets include user specified useful 

y2 packets that contain user defined program identification (PID) information. 

Q ; 15 * The device of claim 1 1, wherein the step of performing rate adjustment includes 

\!*2 a) re-encoding a selected channel, b) removing stuffing bytes, and c) removing user-defined un- 

'^3 useful packets, and the step of combining and transmitting includes combining and transmitting 

4 the re-encoded channel. 

1 1 6. A method for processing video data, the method including the steps of: 

2 examining a plurality of channels of bit streams each having a bit rate; 

3 determining whether a sum of the bits rates for the plurality of channels is equal to a 

4 channel capacity; 

5 if the sum of the bits rates for the plurality of channels is equal to a channel capacity, 

6 scheduling data for the plurality of channels for transmission; and transmitting 
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data for the plurality of channels. 

17. The method for processing video data of claim 16, further comprising the steps of 
determining whether the sum of the bits rates for the plurality of channels is greater than 

the channel capacity; 
if the sum of the bits rates for the plurality of channels is greater than the channel 
capacity then performing the steps of: 
selecting a channel; 

determining whether the selected channel has a bit rate greater than the minimum 

bit rate for the channel; and 
reducing the bit rate for the channel if it is determined that the selected channel 

has a bit rate greater than the minimum bit rate for the channel. 

18. The method for processing video data of claim 17, further comprising the steps of: 
determining whether there is another channels having a bit rate greater than its minimum; 

and 

performing rate conversion if there are no channels having a bit rate greater than its 
minimum. 

19. The method for processing video data of claim 16 5 further comprising the steps of 
determining whether the sum of the bits rates for the plurality of channels is less than the 

channel capacity; 

if the sum of the bits rates for the plurality of channels is less than the channel capacity 
then performing the steps of: 
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selecting a channel; 

determining whether the selected channel has a bit rate is close to the minimum 

bit rate for the channel; and 
increasing the bit rate for the channel if it is determined that the selected channel 

has a bit rate close to the minimum bit rate for the channel 

20. The method for processing video data of claim 19, further comprising the steps of: 
determining whether there is another channels having a bit rate less than its maximum; 

and 

adding null packets to the transmission if there are no channels having a bit rate less than 
its maximum. 

21 . The method for processing video data of claim 20, wherein the step of adding 
extra packets, instead of adding the null packet, adds packets having useful data that the include 
user defined program identification information (PID) for the purpose of sending opportunistic 
data. 

22. The method for processing video data of claim 16, wherein at least one of the 
plurality of channels of bit streams is a data stream that is buffered and not recoded. 

23. The method for processing video data of claim 1 8, wherein the step of reducing 
the bit rate for the channel includes the steps of dropping a B-frame and replacing it with a new 
B-frame which only repeats the previous frame. 

24. The method for processing video data of claim 1 6, wherein the step of scheduling 
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2 data for the plurality of channels for transmission includes the step of selecting packets for 

3 transmission from the plurality of channels based on the channel bandwidth, and the respective 

4 rates of the plurality of channels. 

1 25 . The method for processing video data of claim 24, wherein the step of selecting 

2 packets for transmission is performed by selecting a number of packets from each channel 

3 according to the following equation: 

4 PacketPerTable (I) = (ScheduleTableSize * Rate(i))/(TotalBandwidth). 
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1 System and Method For Multiple Channel Statistical Re-Multiplexing 

2 

3 Abstract of the Disclosure 

4 A system and method for multiple channel statistical re-multiplexing preferably 

5 comprises a plurality of encoders each coupled to a respective channel and producing a 

6 compressed channel, a statistical re-multiplexer and a transport medium. The statistical re- 

7 multiplexer preferably has a plurality of inputs and an output for combining the input 
□ 8 compressed channels into a single output bit stream. The statistical multiplexer further 
jilP comprises a plurality of de-multiplexers and a scheduler & multiplexer having a plurality of 
mo buffers, a scheduling table and a controller. Each of the buffers receives and stores compressed 
J/ data from a respective encoder for a respective channel. The output of each buffer is coupled to a 
Q2 respective de-multiplexer that re-encodes the compressed channel in response to control signals 
j|3 from the scheduler & multiplexer. The scheduler & multiplexer receive the re-multiplexed 

[|4 streams from the re-multiplexers, and combine them into a single stream that matches the 

15 bandwidth of the physical transport medium. The present invention also includes a method for 

16 performing statistical re-multiplexing including the steps of: performing bit stream analysis; 

1 7 determining a sending rate for each channel; determining whether the combined bandwidth 

18 requirement of all the channels exceeds the channel capacity; performing rate adjustment by re- 

19 multiplexing the channels if the combined bandwidth requirement of all the channels exceeds the 

20 channel capacity; scheduling the channels for transmission, combining the channels and 

21 transmitting the combined channels over the transport medium. 
22 
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